Techniques in Computational Stochastic Dynamic Programming

Excerpted Section A. MARKOV CHAIN APPROXIMATION Another approach to finite differences is the well developed Markov Chain Approximation (MCA) of Kushner [3, 4]. Recent developments are surveyed and further advanced by Kushner [5], and by Kushner and Dupuis [6], with special attention to methods for jump and reflected diffusions. This method applies a Markov chain approximation to continuous time, continuous state stochastic control problems by renormalizing finite differences forms as proper Markov chain transition probabilities. These transition probabilities arise when deriving finite difference versions of the dynamic programming equation. An important advantage of this method is that the Markov chain approximation facilitates convergence proofs for the numerical methods in terms of probabilistic arguments. Probabilistic interpretation of the approximation is a major motivation for the formulation of this method. Here, the MCA method is given a formal presentation, in the spirit of the SDP notation and formulation to facilitate comparison. The reader should refer to the above references for the greater detail, especially Kushner and Dupuis [6] for a multitude of variations and convergence proofs. 1. MCA Dynamic Programming Model Formulation Consider the stochastic diffusion without Poisson jumps governed by the stochastic differential equation (SDE) (1) where the notation is the same as in the full paper [2]. It is assumed that drift and Gaussian coefficient are bounded, continuous and Lipshitz continuous in the state , while is uniformly so in the control . Further, let the expected cost objective functional be